Designfax weekly eMagazine

Subscribe Today!

Archives

View Archives

Partners

Manufacturing Center
Product Spotlight

Modern Applications News
Metalworking Ideas For
Today's Job Shops

Tooling and Production
Strategies for large
metalworking plants

Why driving in snow is such a challenge for AI sensors

Pavement can be hard to find on some winter roads. Sensor technology and image processing could help autonomous vehicles better navigate snowy conditions. [Credit: Sarah Atkinson/Michigan Tech]

 

 

 

 

A major challenge for fully autonomous vehicles is navigating bad weather. Snow, especially, confounds crucial sensor data that helps a vehicle gauge depth, find obstacles, and keep on the correct side of the yellow line -- assuming it is visible.

Averaging more than 200 in. of snow every winter, Michigan's Keweenaw Peninsula is the perfect place to push autonomous vehicle tech to its limits. In two papers presented at SPIE Defense + Commercial Sensing 2021 in April, researchers from Michigan Technological University discuss solutions for snowy driving scenarios that could help bring self-driving options to cities like Chicago, Detroit, Minneapolis, and Toronto.

Just like the weather at times, autonomy is not a sunny or snowy yes-no designation. Autonomous vehicles cover a spectrum of levels, from cars already on the market with blind-spot warnings or braking assistance, to vehicles that can switch in and out of self-driving modes, to others that can navigate entirely on their own. Major automakers and research universities are still tweaking self-driving technology and algorithms. Occasionally, accidents occur, either due to a misjudgment by the car's artificial intelligence (AI) or a human driver's misuse of self-driving features.

Sensor fusion
Humans have sensors, too. Our scanning eyes, our sense of balance and movement, and the processing power of our brain help us understand our environment. These seemingly basic inputs allow us to drive in virtually every scenario, even if it is new to us, because human brains are good at generalizing novel experiences.

In autonomous vehicles, two cameras mounted on gimbals scan and perceive depth using stereo vision to mimic human vision, while balance and motion can be gauged using an inertial measurement unit. However, computers can only react to scenarios they have encountered before or been programmed to recognize.

Since artificial brains aren't around yet, task-specific artificial intelligence (AI) algorithms must take the wheel, which means autonomous vehicles must rely on multiple sensors. Fisheye cameras widen the view, while other cameras act much like the human eye. Infrared picks up heat signatures. Radar can see through the fog and rain. Light detection and ranging (lidar) pierces through the dark and weaves a tapestry of laser beam threads.

"Every sensor has limitations, and every sensor covers another one's back," said Nathir Rawashdeh, assistant professor of computing in Michigan Tech's College of Computing and one of the study's lead researchers. He works on bringing the sensors' data together through an AI process called sensor fusion.

"Sensor fusion uses multiple sensors of different modalities to understand a scene," he said. "You cannot exhaustively program for every detail when the inputs have difficult patterns. That's why we need AI."

VIDEO: Drivable path detection using CNN sensor fusion for autonomous driving in the snow. [Credit: Michigan Tech]

Rawashdeh's Michigan Tech collaborators include Nader Abu-Alrub, his doctoral student in electrical and computer engineering, and Jeremy Bos, assistant professor of electrical and computer engineering, along with master's degree students and graduates from Bos' lab: Akhil Kurup, Derek Chopp, and Zach Jeffries. Bos explains that lidar, infrared, and other sensors on their own are like the hammer in an old adage. "'To a hammer, everything looks like a nail,'" quoted Bos. "Well, if you have a screwdriver and a rivet gun, then you have more options."

Snow, deer, and elephants
Most autonomous sensors and self-driving algorithms are being developed in sunny, clear landscapes. Knowing that the rest of the world is not like Arizona or southern California, Bos's lab began collecting local data in a Michigan Tech autonomous vehicle (safely driven by a human) during heavy snowfall. Rawashdeh's team, notably Abu-Alrub, poured over more than 1,000 frames of lidar, radar, and image data from snowy roads in Germany and Norway to start teaching their AI program what snow looks like and how to see past it.

"All snow is not created equal," Bos said, pointing out that the variety of snow makes sensor detection a challenge. Rawashdeh added that pre-processing the data and ensuring accurate labeling is an important step to ensure accuracy and safety. "AI is like a chef -- if you have good ingredients, there will be an excellent meal," he said. "Give the AI learning network dirty sensor data, and you'll get a bad result."

Low-quality data is one problem, and so is actual dirt. Much like road grime, snow buildup on the sensors is a solvable but bothersome issue. Once the view is clear, autonomous vehicle sensors are still not always in agreement about detecting obstacles. Bos mentioned a great example of discovering a deer while cleaning up data gathered locally. Lidar said the blob was nothing (30% chance of an obstacle), the camera saw it like a sleepy human at the wheel (50% chance), and the infrared sensor shouted WHOA (90% sure that is a deer).

Getting the sensors and their risk assessments to talk and learn from each other is like the Indian parable of three blind men who find an elephant: each touches a different part of the elephant -- the creature's ear, trunk, and leg -- and comes to a different conclusion about what kind of animal it is. Using sensor fusion, Rawashdeh and Bos want autonomous sensors to collectively figure out the answer -- be it elephant, deer, or snowbank. As Bos puts it, "Rather than strictly voting, by using sensor fusion we will come up with a new estimate."

While navigating a Keweenaw blizzard is a ways out for autonomous vehicles, their sensors can get better at learning about bad weather and, with advances like sensor fusion, will be able to drive safely on snowy roads one day.

Source: Michigan Technological University

Published July 2021

Rate this article